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ABSTRACT 





N items have been tested . Attempts are made to correct those failure 
modes that occur; the probabilities that these attempts are successful are 



assumed known . The conditional distribution of the resulting true reliability 
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is derived and used to construct observable limits which will include the true 
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reliability with a known probability . ' > 1 i“ ; ‘ ; 
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UNCLASSIFIED 



CONDITIONAL DISTRIBUTION OF TRUE RELIABILITY 
AFTER CORRECTIVE ACTION 



1 , Introduction 






This paper considers a situation in which N items of a particular sort 
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are tested. N of these items work correctly and N-N fail. The items 
o o 
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that fail are scrutinized and for some (or all) the cause of failure is deter- 
mined. Corrective action then is taken to remove those failure modes . . ' . 
that are observed in the N tests . The corrective actions taken may or 
may not be successful (in removing their respective failure modes from , 
subsequent items); no further items of a corrected design are tested. It 
is assumed that the probability of correcting. any observed failure mode . : 

is known. . '• V' 

The value of the true reliability of a system after attempts have been 
made to remove the failure modes that occur in N independent tests of 

. /• • * 

the system is a quantity of vital interest. This true reliability is, of 
course, a random variable, a quantity whose value depends upon the out- 
come of a chance experiment. Since no tests of the system are assumed 

, .* *V “ 

to have been made after the correctional attempts, there is available no 
direct experimental evidence on the reliability value finally attained . How- 
ever, assuming we have available the results of the N tests of the system 
and that we know the probability that any occurring failure mode is corrected, 
it is possible to construct the (conditional) distribution of the true relia- 
bility, and to explore this distribution for possible meaningful statements 
that can be* made . 
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The paper "Estimating Reliability After Corrective Action" defines p 






as a measure of current reliability and then goes on to discuss and compare .V- 
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estimators of the average value of p Q , called the mean reliability . It is 
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stressed by the authors that the paper is concerned with estimators whose 
properties are to be studied prior to the N tests having been performed, 
thufl Justifying the concern about mean reliability, an average taken over v 
all possible outcomes of the N tests An approach such as theirs is 
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undoubtedly of interest in the early phases of development of particular 
types of item . 

It would seem that an equally important problem can be phrased in a 
rather different way. Since the final estimator of achieved reliability 
(subsequent to the corrective action) is reasonably a function of the number 
of failures of various sorts that occur in the N tests, the conditional 
distribution of true reliability (conditional upon the outcome of the N 
, tests) might be expected to yield some valuable information. This condi-^ 
tional distribution is exactly known (in some senses) at the time the actual 
estimator of achieved reliability is to be evaluated . This paper gives the • 
conditional distribution of true reliability. 

This conditional distribution of true reliability can be used to con- 
struct a conditional confidence interval for the achieved true reliability, 
based upon the results of the N tests . This conditional confidence 
interval gives a number, call it Z q , such that the probability is at least 
1-cr that the true reliability is as large as Z q ( or larger) . As such, it 
reflects both the information contained in the results of the N tests and 
the assumed knowledge regarding ability to correct observed failure modes . 

As will be seen , the conditional confidence interval statements that 
are given involve some generally described functions. It is expected, but, 
not yet investigated, that for some very special types of these functions 
relatively tight probability, statements can be made. That is, by adjusting - 
the functions involved, it seems probable that the probability of exceeding 
Z q might be made quite close to 1-or, rather than possibly being quite a 
bit larger than 1 - a . 

2 . Conditional distribution of true reliability . 



N systems are to be tested. Each system either operates successfully 
(with probability ^) or it fails to operate successfully (with probability 
1~P q ) , A system which does hot operate successfully can fail according 
to any one of K different' modes (with - probabilities of occurrence equal 



to q A ; i 1 1 2 1 « • • , K ) . 



Thus 
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i=l 

In writing the conditional distribution of true reliability, we can choose 
to represent either all K possible failure modes or to represent only those 
R modes that actually occur. If we make the first of these two choices, 

then in the N system tests we may not observe all K modes ♦ Letting a. / 

til ^ 

be the known probability of correcting the i failure mode, i=l, 2, . . , ,K# 

the probability mass function of Z, the true reliability, is as given in . 

th \ 

Table 1, the variable x. is 1 if the i failure mode is observed one or 

i • . 

more times in the N tests and is 0 otherwise, 

TABLE 1 






^ • 



V . . i 

» .V ! \ 



■ rv ' i. 



/*» ; 



, . .t . • y 



.T-. * 



■.<« * ■ 



: ' z. 

• i 


P(Z=Z.). ; V-' , 


v*> ? p- 


7 r (1-a) i ■ •• . - .r. 

>.i= 1 . :■ "'>v - f:x y.;' 


•..V 1 •••: ... . “o 1 .-•••. : V ' ; •- 


Vi i2 2 (1 r a i ) 1 .•••; 


V q 2'' 


a 2 x 2 : 


*• ' v • ■ * * -r. ’.♦> 

. Po +q k 


k ' tv 

.^(l-a,) i a k x k . . . 


v Po +q l +q 2 

' * # 


✓ % : ' l t • • \ % ■ • ' / • 

Vi _ V 2 ,1 3 <‘- a i ) . Xl . ■ 


# 




, p 0 +q k-i + V 

• 1 • * 

4 *' ' . • * 


’* \ y - ;•* , 1*. •' ■ 4 * ' . 


t 

q l +q 2 + .- V ' +q liv ; 1 

* *• • 


. ii! |- { -rk • 

- v«' > • . • . ' 

TT a i X. .<;« ; . - . ... • ' 

••• : < ■ 

*. .'j - 3. 




■ '• ■' . ' ' + 


' ' ’ ' ' • ' ’ •S, ( 









;.V>. h. 



r\\ *,'• P 
.V* •*,’ 

r'YW* ■■ 









This probability mass function is a little unusual in that we are assuming 

we do not know the values the time reliability takes on, since they are 

functions of p and the q 's but we do know the probabilities with 
o i 

Which these unknown values occur. 

We can easily derive the moment generating function for this distri- 
bution by examining the terms we get in expanding such a function as 

< a l +b l> (a 2 + b 2 ) ••• <a k + b k > - 
If we form the sum defining 

:* z (t) = E(e tZ ) 

this sum contains all the terms (and no others) in the expansion of 



‘ tf V 1 (w tqi+ '‘V 1 )'. 



Thus , the moment generating" function for Z (in this formulation) is 



' t z (t) = e tp o (a.xje tq i + '(1**^) *i ) 



Evaluating ^ (0) and f„(0) we find 
Z " 

' . K / 
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i=l 



;• ’■ * . . , ' . 



E [ (z-E® J ] - l (1-a,) x.q^ 



(Both of these expectations are conditioned upon the results of the N 
, * * 

tests) . Thus , the measure of true reliability (p Q ) mentioned in the paper 
by Corcoran, Weingarten and Zehna is in fact the mean value of the 
conditional distribution of final reliability,* If we average this mean value 



over all possible experimental outcomes, we arrive at mean reliability, 
the quantity they were estimating . Especially since the actual estimation 
takes place only after the N tests have been performed, it would seem 



reasonable to try to estimate p^ (or some other property of the conditional 
distribution) rather than something influenced by events that are known not 
to have occurred, when the estimation actually takes place. The second, 
simpler (and more natural) representation of the conditional distribution of 
true reliability is given by considering only those failure modes that 
actually occur in the N system tests (rather than considering all possible 
failure modes that could occur) Let Z be the true reliability (after the . 
N tests and corrective action) and assume that R failure modes occur. 
Then the conditional probability mass function for Z is as given in table 
one above, where.we set = 1, for i=l , 2, * .., R, and set x^ = 0, for ^ 



i= R+l, * . . , K 



Then the moment generating function for Z (in this revised situation) is 




The mean and variance of Z are ; ? 
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We would like to know the actual value that the true reliability has ? 
taken on, after the corrective action has been completed; this value is 
a random variable whose conditional distribution is given above. Since 
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rthe actual value cannot be observed, the natural substitute to turn to is 
estimation of various properties of the conditional distribution of Z . 
The question of point estimation in general will be addressed first and 
then the idea of interval estimation of Z will be investigated . 

3. Point Estimation 




: 



The most frequently estimated parameter of a probability distribution 
is undoubtedly its mean. Accordingly, if we were to make a point esti- 
mate of some property of the conditional distribution of Z, the most 
logical candidate to consider first might be the mean of the distribution. 
We have available the results of the N system tests for making such an 
estimate and thus could consider any estimator which is a function of 7- 
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N , N, , N« ;'N n . The estimator 
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is in fact an unbiased estimate of p q , averaging over all possible out- 



comes of the experiment. However, since we are committed here to con- 
sidering only properties of the conditional distribution of Z , it would 
be no more than reasonable to consider the properties of estimators 
averaging only over those experimental outcomes such that the distri- 
bution of Z remains unchanged . If we do not take only such conditional 
averages into account, then since the conditional distribution of Z may < 
change so may the quantity to be estimated and we are effectively esti- 
mating something different from what was intended. The conditional 
distribution of Z remains unchanged so long as the failure modes 
observed are unchanged, no matter how many times they are observed. 
Thus we would like to average only over those experimental outcomes 
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such that the observed -failure modes still occur and we are led : 
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to defining the eventA(on the space of all possible outcomes in the N 



system tests) to be the set of outcomes such that N >0 , N >0 , . . . , N > 0 , 

X Ct i\ 



where failure modes. 1, 2 . . . , R are the ones that are observed in the tests 
If A occurs , the conditional distribution of Z is fixed. In discussing 



• properties of point estimators we shall make use of conditional expecta- 






tions only, in accordance with the discussion just given, 
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As was mentioned above, the estimator //!• . 
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' is an unconditionally unbiased estimator of p (since E(N,) = Nq,) . How- •* / - -y 

o i i . 

• ever, it is not a conditionally unbiased estimator, as will be shown below, V i; ,' 
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For simplicity, suppose that we perform N system tests and observe 
only 2 failure modes . Define 
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Without a great deal of difficulty we find (again k= N-i-j) 
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d 'X' \Then, combining the above results we find 
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It would appear to be a hopelessly complex job to add a quantity to p. V * ; . ‘ 

' to correct for its bias (in estimating p conditionally); possibly it could , * . ' 
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not be done. Since conditional unbiasedness seemed so difficult to 



achieve, the investigation of point estimators has initially stopped at 
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this point. More thought should be given to what parameters of the 
conditional distribution are of particular interest before much of an 
attempt is made to construct "good" estimators of them. In passing, in 
the case of R failure modes occurring, it is conjectured that 
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These values can (possibly) most easily be derived by taking derivatives ' • , 

of P(A) with respect to the parameters p , q. , . . . q_ and solving the 

O 1 K 

resulting system of equations for the conditional expectations of interest. : 



4 . Interval estimation. 



./'..O'; , An apparently more fruitful avenue of investigation is that of interval ' 

estimation of the true reliability Z . Since we know the probability distri- • . 

• • ■; - A 

• I* <• 

. "• bution of Z it should be easy in theory to find a point b such that j 
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P ( Z > b) ^ 1 - a 



i i 



and thus the interval (b, 1) would constitute a 100 (1-a) % confidence • .• XjH- 

,r.v- 

interval for Z . The reason that we cannot immediately find the value b, 
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of course, is we do not in general know p , q , . . ., q and thus do -v,yy*. 
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not explicitly know the values (nor even the ranking of them) that the 

. * 'i ; ’ 

true reliability will take on. However, there are some interesting lines 
# # 
of reasoning that can be explored, leading almost to confidence intervals 

i 

for Z (one approach leads exactly to confidence intervals of the above ", 

• . / ; 

sort) . We shall consider two separate cases, the first assuming no 
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information whatsoever about p ,'q ,.. . ,'q and the second, assuming 
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that we have available a ranking in order of magnitude of q , q. , . . . q , 

1 c* i\ 

determined independently from the current set of tests . Within each of 
these we shall discuss both conditional and unconditional confidence 
intervals for Z . 



In the event that p , q, , q 9 . . . , q are completely unknown, the 
values and ranking of values that the true reliability takes on are unknown 
However, due to the definition of the probabilities of successes and 
failures occurring there is a natural partial ordering that can be exploited 
to construct conditional interval statements about Z , Thus, since' 
p :s 0, q, * 0, >0, we know that 

O 1 * K * , * 

i • • . i • 

K + Si < + % + % < . . < p n + q, + . . . + q D 

0,01012 Ol R 

P • < P_ + <3o < P„ + q 9 + q, < . . . < p rt + q, + . . . + q_ 

00 4 021 Oi R, 

and so on. In fact, it can be shown that the true reliability Z may take 
on any of 2 (conceivably) distinct values if R failure modes occur.- 

p + q. will precede 2^ *-l of these, for all i, and is indeterminate with 

® f R“1 r- 

respect to 2 of them. More generally, p plus r q.'s will succeed 2 -1 

n O 1 

,R-r 



values , precede 2 -1 values and is indeterminate with respect to the' : 

remaining (2^ r *-l) (2 r -l) values. , . : : 

This natural partial ordering can be used to make statements such 



as 
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etc. It if should happen that one of these right hand sides is equal to 
a confidence level of interest, then we alihost have a confidence interval . 



Almost, because we still do not know p and the q.'s and thus would have 

o 1 

to estimate them in order to have some idea where the minimum value in- 
volved might fall. But in making use of such an estimate we would no 
longer have an exact probability statement about the interval containing Z. 

However, if we think of the compound experiment consisting of the N 
system tests and then the corrective action, the reasonable probability 
measure to use for the compound experiment is constructed by multiplying 
together the measure that applied to the N system tests , times the conditional 
measure applicable to the corrective action taken (conditional only in the sense 
that the experimental outcome dtermines the corrective action to be taken) . 
Using this multiplicative rule enables us to make statements such as 
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where A is the event defined above. The results of the N system tests 

i ■ * ’ * * 

!>, ' are available to evaluate f (N , N, , . . ., N_ ) occurring in the first of these 

: , V- a o l R 

two probability statements (this is just a conditional multinomial probability .<>. 

- statement) while reasoning such as the preceding may be used to evaluate 

■,'Vo 

.'*0 ' the second. Their product gives us a statement that is somewhat similar 

to a conditional confidence interval; this product can be used to construct 

; <• • 

a lower bound confidence interval as follows. Let g be a function of the • : : 
unknown parameters , let f be a function of the observable random variables 
N©/ Nj, . . N r , and let A be the event that > 0, > 0, ...» ^>‘0 . 

' - Then a conditional confidence interval for Z would be a statement such as 
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The statement we are able to make is 

P {Z>g, g>f | A) = P (Z.>g | A) P (g > f | A | ' 

\ «... 
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* .* -.. 


= 1 'r!:?:' 

Since the set relationship { Z> / g, g> f } c { Z>f} must hold, we can , 
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K '' • / # ' 4 
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say . .• • r ! - 

P (Z > f | A) ^ P ( Z > g, g>£ | A) and thus get a lower bound 
on the probability that an observable interval contains Z . ' 


1 ^ */ . ’ ' 
c • 7 
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Unconditional confidence intervals would possibly not be of much 
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interest, for many of the same reasons given previously regarding expecta- ' 
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tions averaged over all possible experimental outcomes . Since we would ■' 
not be constructing the interval, presumably , until alter the N. system . V v ' 
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tests are completed, we would be more interested in the conditional V. 


... 


measure of Z than we would be in the unconditional measure . Even if we 
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had the interest, it would appear to be impossible to construct an uncondi- 
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tional confidence interval for Z , since the probability that Z exceeds • 
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some function f of the observable random variables N , N. , . .., N„ . • > . 

o 1 R 


* .V. * V 
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is going to depend upon what values of the form p , p +q, , p +q_, etc., 

o 0 1 0 2 
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are exceeded by f; given no information about p ., q, , . . ., q„, we are 

O 1 R VV.- 1 ‘ .. 

not able to determine which of these values are and which are not exceeded 
by f . Thus we cannot logically derive an unconditional measure of the ' • v" 


\ * . < 
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probability that Z exceeds f. , ; • ’ . •- ' ... (v r\< 
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Suppose we are given that , *.’• 

V ' ' • * *' 
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q i >q 2 > > q R,. -»• • • ' 

Then we would find that again 'Z can take on any of 2^ (conceivably) 

distinct values, but now p +q, is indeterminate with respect to only 

R~1 ® ^ r_ 2 ■ ■ - \ n 

2 -R of them, p +q is indeterminate with respect to 2 -R+l of them, V ; , 

etc.; in general, Is indeterminate with respect to 2 — (R— J) -1 of TO.'y 

the other values , for 1*1,2,... /R-i . The position of p +q is completely • 
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determined with respect to all the possible values. Similarly, we find 

that the position of p + q +q„ is andeterminite with respect to 

oil 

^R-3 ” (R— 3) -1 values , p Q -»-ci^+q^ is lindeterminate with respect 

to 2^"*^- ** va l ues * an< ^ so on * The assumed ranking of • 

the q^'s thus leads to a more definite ranking of the values of Z , which 
in turn would imply that we have a much wider choice of values for state- 
ments such as 



\ 



P ( Z>g | A) , 

where g is some function of the unknown parameters . Thus we could 
construct confidence intervals for a much larger choice of probabilities . 
Exactly the same type of reasoning as that outlined above would lead to 

4 

lower bounds, on the conditional probabilities that particular intervals 

V 

cover Z . Unconditional statements would appear as impossible of attain- 
ment here as in the preceding case, for exactly the same sort of reasons . 



5 . Topics for further' research . 

The conditional multinomial probability statement mentioned in section 
4 above may or may not be a simple matter to derive . The feasibility and 
explicit method of attaining such a statement should be spelled out in 
detail . 

The assumption that the a^'s are. known (the probabilities of removing 
observed failure modes) may not be realistic. The affects of these quanti- 
ties being unknown should be detailed. In many cases it would seem 
plausible to assume p ,q, , . . . ,q known and a , a , . . .,a unknown; the 
affects of this change should be investigated . 

It might prove profitable to. replace the probabilities of correcting 
observed failure modes by the reliability of the replacement part (or design 
change) . That is, if a design change is made then a^ is essentially the 
probability that the reliability of the change is 1; 1-a is the probability 

i f . 1 . . . 



that the reliability is less than 1 (but no indication of how much less 



than 1) . Rather than include the parameters a,, in the model it might 






prbve better to use the parameters b. , defined to be the reliability 

1 * 

of the changed item or portion of the design. This would, of course, , 

. allow final reliability after corrective action to decrease rather than only 
* to increase as in the present model . 

The whole basic model for reliability growth might be better if 
structured in some different manner. The suggestion made in the last J 
V; paragraph is one possible restructuring; undoubtedly many others could 

be brought to mind. The restructuring should keep in mind simplicity of y-;S:v - 
‘ approach, ease of computation and estimation and yet remain as realistic 
: as possible. 
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